Bundling night vision and other driver assistance systems (DAS) using near infra red (NIR) illumination and a rolling shutter

ABSTRACT

A system mountable in a motor vehicle. The system includes a camera and a processor configured to receive image data from the camera. The camera includes a rolling shutter configured to capture the image data during a frame period and to scan and to read the image data into multiple image frames. A near infra-red illuminator may be configured to provide a near infra-red illumination cone in the field of view of the camera. The near infrared illumination oscillates with an illumination period. A synchronization mechanism may be configured to synchronize the illumination period to the frame period of the rolling shutter. The frame period may be selected so that the synchronization mechanism provides a spatial profile of the near infra-red illumination cone which may be substantially aligned vertically to a specific region, e.g. near the center of the image frame.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims benefit from U.S. patent application Ser.Nos. 61/421,424 filed on 9 Dec. 2010 and 61/408,617 filed on 31 Oct.2010 by the same inventors the disclosures of which are incorporatedherein by reference.

BACKGROUND

1. Technical Field

The present invention relates to driver assistance systems in motorvehicles, and in particular to providing a system and a method for nightvision using a near infra red (NIR) Illumination and a rolling shutterfor driver assistance systems (DAS) applications.

2. Description of Related Art

During the last few years camera based driver assistance systems (DAS)have been entering the market; including lane departure warning (LDW),automatic high-beam control (AHC), traffic sign recognition (TSR) andforward collision warning (FCW).

Lane departure warning (LDW) systems may be designed to give a warningin the case of unintentional lane departure. The warning is given whenthe vehicle crosses or is about to cross the lane marker. Driverintention is determined based on use of turn signals, change in steeringwheel angle, vehicle speed and brake activation. There may be variousLDW systems available. One algorithm for lane departure warning (LDW)used by the Applicant/assigne (Mobileye Technologies Ltd., Nicosia,Cyprus, hereinafter “Mobileye”) of the present application is predictivein that it computes time-to-lane crossing (TLC) based on change inwheel-to-lane distance and warns when the time-to-lane crossing (TLC) isbelow a certain threshold. Other algorithms give a warning if the wheelis inside a certain zone around the lane marker. In either case,essential to the lane departure warning system is the lane markerdetection algorithm. Typically, the lane markers may be detected in thecamera image and then, given the known camera geometry and cameralocation relative to the vehicle, the position of the vehicle relativeto the lane is computed. The lane markers detected in the camera imagemay be then collected over time, for instance using a Kalman filter.Wheel-to-lane marker distance may be given with an accuracy of betterthan 5 centimeters. With a forward looking camera, wheel-to-lane markerdistance is not observed directly but is extrapolated from the forwardview of the camera. The closer road markings may be observed, lessextrapolation is required for determining wheel-to-lane marker distanceand more accurate estimates of wheel-to-lane marker distance may beachieved especially on curves of the road. Due to the car hood and thelocation of the camera, the road is seldom visible closer than sixmeters in front of the wheels of the car. In some cars with longerhoods, minimal distance to visible road in front of the car is evengreater. Typically the lane departure warning system of Mobileye workson sharp curves (with radius down to 125 m). With a horizontal field ofview (FOV) of 39 degrees of the camera, the inner lane markers may bestill visible on curves with a radius down to 125 meters. In order tocorrectly perform lane assignment on curves, lane markings may bedetected at 50 meters and beyond. With a horizontal field of view (FOV)of 39 degrees for the camera, a lane mark of width 0.1 meters at 50 mdistance corresponds in the image plane to just under two pixels wideand can be detected accurately. The expectation from the lane departurewarning systems is greater than 99% availability when lane markings maybe visible. Expectation with 99% availability is particularlychallenging to achieve in low light conditions when the lane markingsmay be not freshly painted (have low contrast with the road) and theonly light source is the car halogen headlights. In low lightconditions, the lane markings may be only visible using the highersensitivity of the clear pixels (i.e. using a monochrome sensor or ared/clear sensor). With the more powerful xenon high intensity discharge(HID) headlights it is possible to use a standard red green blue (RGB)sensor in most low light conditions.

Traffic sign recognition (TSR) modules may be designed typically todetect speed limit signs and end-of-speed limit signs on highways,country roads and urban settings. Partially occluded, slightly twistedand rotated traffic signs may be preferably detected. Systemsimplementing traffic sign recognition (TSR) may or should ignore thefollowing signs: signs on truck/buses, exit road numbers, minimum speedsigns, and embedded signs. A traffic sign recognition (TSR) module whichfocuses on speed limit signs does not have a specific detection rangerequirement because speed limit signs only need to be detected beforethey leave the image. An example of a difficult traffic sign to detectis a 0.8 meter diameter traffic sign on the side of the road when thevehicle is driving in the center lane of a three lane highway. Furtherdetails of a TSR system is disclosed by the present assignee in patentapplication publication US20080137908.

A typical automatic headlight or high/low beam control (AHC) systemdetects the following conditions and switches from high beams to lowbeams: headlights of oncoming vehicles, taillights of precedingvehicles, street lights or ambient light indicating that high beams maybe not required and a low vehicle speed. The host vehicle lights may beswitched back to high beams when none of these conditions exist (oftenafter a specified grace period). One approach for detecting taillightsis to compare images from two sensors: one with a red filter and thesecond with a cyan filter. The cyan filter responds to non-red lightsources and will give zero response to red light. By comparingcorresponding pixels from two imaging sensors one can detect the colorof the light source. The number of pixels of each color above a certainintensity is counted and if the count is above a threshold the systemsswitches to low beams. The use of color filters with imaging sensors maypreclude the simultaneous use of the same image frames for other driverassistance applications.

A second approach for automatic high-beam control (AHC) uses an RGBsensor to give better color differentiation. Typical light sources canbe located in the full CIE color space as defined by the InternationalCommission on Illumination. This approach distinguishes between green,yellow and red lights. A powerful green traffic light is not confusedwith an oncoming vehicle. Since a single sensor with a color mosaicfilter i.e. Bayer pattern mosaic is used, the lens is defocused so as tospread a light source over multiple pixels. The use of the color mosaicfilter reduces both the effective image sensor resolution (by 50%) andthe intensity response (to less than one third). The color mosaic filtermay preclude the use of the same sensor for traffic sign recognition(TSR) or lane departure warning (LDW) because of the intensity responsepenalty.

Ego-motion estimation is disclosed in U.S. Pat. No. 6,704,621 by Stein.Image information is received from images recorded as the vehicle movesalong a roadway. The image information is processed to generate anego-motion estimate of the vehicle, including the translation of thevehicle in the forward direction and the rotation.

Some driver assistance systems may rely on changing camera exposureparameters (e.g. aperture, exposure, magnification). The use of a colorcamera equipped for instance with an RGB (red/green/blue) filter and aninfrared filter achieves good spectral separation for detectingtaillights or brake lights but reduces imaging sensitivity by a factorof six or more. A reduction in sensitivity by such a factor has anadverse impact on other vehicle control application such as LDWperformance in dark scenes. The presence of an infrared filter alsonegates the use of the camera as a near infrared sensor forapplications, such as pedestrian detection. Thus, a brake lightdetection system which uses color or spectral analysis in the capturedimages may be less compatible with other driver assistance systemswithout sacrificing performance.

The core technology behind forward collision warning (FCW) systems andheadway distance monitoring is vehicle detection. Assume that reliabledetection of vehicles in a single image a typical forward collisionwarning (FCW) system requires that a vehicle image be 13 pixels wide,then for a car of width 1.6 m, a typical camera (640×480 resolution and40 deg FOV) gives initial detection at 115 m and multi-frame approval at100 m. A narrower horizontal field of view (FOV) for the camera gives agreater detection range however; the narrower horizontal field of view(FOV) will reduce the ability to detect passing and cutting-in vehicles.A horizontal field of view (FOV) of around 40 degrees was found byMobileye to be almost optimal (in road tests conducted with a camera)given the image sensor resolution and dimensions. A key component of atypical forward collision warning (FCW) algorithm is the estimation ofdistance from a single camera and the estimation of scale change fromthe time-to-contact/collision (TTC) as disclosed for example in U.S.Pat. No. 7,113,867.

BRIEF SUMMARY

According to the present invention there is provided a system mountablein a motor vehicle. The system includes a camera and a processorconfigured to receive image data from the camera. The camera includes arolling shutter configured to capture the image data during a frameperiod and to read the image data into multiple image frames, forinstance alternately line by line. A near infra-red illuminator may beconfigured to provide a near infra-red illumination cone in the field ofview of the camera. The near infrared illumination oscillates with anillumination period. A synchronization mechanism may be configured tosynchronize the illumination period to the frame period of the rollingshutter. The image frames include a long exposure frame and a shortexposure frame. The synchronization mechanism is configured to turn onthe near infra-red illumination during exposure of at least one of themiddle rows of the long exposure frame. The frame period may be selectedso that the synchronization mechanism provides a spatial profile of thenear infra-red illumination cone which may be substantially alignedvertically to a specific region of the image frame.

The processor may include multiple driver assistance modules configuredto perform respective multiple driver assistance functions. A firstdriver assistance module may be configured to use the near infra-redillumination and a second driver assistance module may be configured touse parts of an image that do not have the near infrared illumination.The first driver assistance module may be configured to performpedestrian detection responsive to the near infra-red illumination inthe first multiple image frames. The second driver assistance module mayuse color information from the second image frames to detect headlightsof an oncoming vehicle in the field of view of the camera, to detecttaillights of a leading vehicle in the field of view of the camera, todetect taillights of a leading vehicle in the field of view of thecamera and/or to detect a traffic sign in the field of view of thecamera. The camera may be configured to provide to the processor theimage frames. The image frames may be partitioned into first imageframes for use by the first driver assistance module and into secondimage frames for use by a second driver assistance module. A parameterof the camera may be configurable to alternate respective values betweenthe first and second image frames. The parameter may be the frame periodof the rolling shutter.

The synchronization module may be configured to time the near infra-redillumination to be in the first image frames and to avoid the nearinfra-red illumination in the second image frames.

According to the present invention there is provided a method operablein a system mountable in a motor vehicle. The system include aprocessor, a camera having a rolling shutter configured to capture imagedata during a frame period and to scan line by line to read the imagedata of the image frame and to read the image data into a plurality ofimage frames. The method receives image data from the camera. Nearinfra-red (NIR) illumination may be provided in the field of view of thecamera. The near infrared illumination oscillates at an illuminationfrequency with an illumination period with an illumination duty cycle.The illumination period may be synchronized and to the rolling shutter.The image frames include a long exposure frame and a short exposureframe. The synchronization turns on the near infra-red illuminationduring exposure of at least one of the middle rows of said long exposureframe.

The camera may be configured to provide to the processor multiple imageframes. The image frames may be partitioned into first image frames foruse by a first driver assistance module and into second image frames foruse by a second driver assistance module. The synchronization mayprovide the near infra-red illumination in the first image frames andavoids the near infra-red illumination in the second image frames. Thevalues of a camera parameter may be alternated respectively inalternating first and second image frames. The camera parameter may bethe frame period of the rolling shutter.

These, additional, and/or other aspects and/or advantages of the presentinvention are set forth in the detailed description which follows;possibly inferable from the detailed description; and/or learnable bypractice of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIGS. 1 a and 1 b illustrate a system including a near infra red (NIR)illuminator and a camera mounted in a vehicle, according to an aspect ofthe present invention.

FIG. 2 shows a schematic description of exposure settings with a rollingshutter, according to a feature of the present invention.

FIG. 3 shows schematically graphs of NIR intensity in percent for NIR anilluminator that is on for 15 ms and off for 30 ms, according to afeature of the present invention.

FIG. 4 a shows a NIR illuminator that is on for 22 ms and off for 30 ms,where the on time (indicated by horizontal arrow) aligns with the middlerow of a long exposure frame, according to a feature of the presentinvention.

FIG. 4 b shows an NIR spatial profile overlaid on an image withrespective percentage (%) NIR intensities for two frames, according to afeature of the present invention.

FIG. 5 shows schematically a typical spatial distribution in image spacecoordinates of an NIR illumination cone, according to a feature of thepresent invention.

FIG. 6 shows the NIR spatial profile overlaid on an image, according toan feature of the present invention.

FIG. 7 shows a method, according to an aspect of the present invention.

The foregoing and/or other aspects will become apparent from thefollowing detailed description when considered in conjunction with theaccompanying drawing figures.

DETAILED DESCRIPTION

Reference will now be made in detail to aspects of the presentinvention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The aspects are described below to explain the presentinvention by referring to the figures.

Before explaining features of the invention in detail, it is to beunderstood that the invention is not limited in its application to thedetails of design and the arrangement of the components set forth in thefollowing description or illustrated in the drawings. The invention iscapable of other features or of being practiced or carried out invarious ways. Also, it is to be understood that the phraseology andterminology employed herein is for the purpose of description and shouldnot be regarded as limiting.

By way of introduction, features of the present invention may bedirected to a system that performs multiple Driver Assistance (DAS)applications concurrently on the same hardware platform wherein forinstance at least one of the applications requires near infra red (NIR)illumination and at least one other application requires for instancecolor information. For example, one application might be pedestriandetection at night requiring near infra red (NIR) illumination andanother application might be Automatic Highbeam Control (AHC) whichrequires color information.

Rolling shutter is a method of image acquisition in which each frame isrecorded not from a snapshot of a single point in time, but rather byscanning across the frame, for instance row by row. With a rollingshutter, not all parts of the image are recorded at exactly the sametime, even though the whole frame is displayed at the same time duringplayback. The rolling shutter is in contrast with a global shutter wherethe entire frame is exposed for the same time window.

Since near infra red (NIR) illumination is almost monochrome, all colorinformation from reacted objects may be lost. With a global shutterimage sensor, it may be possible to alternate between illuminated andnon illuminated NIR frames. That is, the near infra red (NIR)illumination is on for some image frames (those used for pedestriandetection for example) and off for image frames (those used for colorbased detection for example). A more complex situation is when a rollingshutter sensor is used. The rolling shutter may be more complex becauserows in one image are actually exposed at the same time as other rowsfor the following frame and there may be no period when it is possibleto switch on the illuminator to uniformly expose all the rows of oneimage and none of the rows of the next image. According to differentfeatures of the present invention, when the rolling shutter sensor isused, an illumination frequency and duty cycle may be used which arecarefully tuned to the camera exposure, combined with precise definitionof the illumination cone.

Pedestrians may be detected up to a distance of 20 m using a standardautomotive CMOS camera such as the Aptina MT9V024 (Aptina ImagingCorporation, San Jose, Calif., USA) under low beams. The detectiondistance increases significantly when using high beams but of course theuse of high beams is limited when there is traffic so as not to blindother drivers. One possible way to increase detection distance is to useNIR illumination. NIR illuminators may be standard halogen highbeamswith a long pass filter or dedicated IR illuminators such 850 nm lightemitting diodes or LEDs. However with NIR illumination all colorinformation from reflective surfaces may be lost, particularly for thecase of narrow band 850 nm LEDs but even if a wide NIR spectrum is used,the NIR response of typical red, green and blue pixels of a typicalcolor sensor is the same for all three channels.

Reference is now made to FIGS. 1 a and 1 b which illustrate a system 16including a near infra red (NIR) illuminator 4 and a camera 12 mountedin a vehicle 18, according to an aspect of the present invention.Illuminator 4 may be mounted in the headlights of vehicle 18 or inproximity of camera 12 to project NIR illumination 4 a forward ofvehicle 18. Camera 12 includes a sensor equipped with a rolling shutter6. Camera 12 images a field of view in the forward direction whichcaptures image frames 204/206 in real time and transfers image frames204/206 to processor 14. Image processor 14 may be used to process imageframes 204/206 simultaneously and/or in parallel and to provide multipledriver assistance functions. By way of example in FIG. 1 b, image frames204/206 may be used to serve forward collision warning (FCW) system 22,lane departure warning (LDW) 21, traffic sign recognition (TSR) 20,automatic headlight control 24 and pedestrian detection warning (PDW)23. Synchronization mechanism 2 provides synchronization between therolling shutter 6 of camera 12 and NIR illuminator 4.

Image processor 14 is used to process image frames 204/206 to detect apedestrian in an image in the forward field of view of camera 12 asrequired for pedestrian detection warning (PDW) 23. Pedestrian detectionwarning (PDW) 23 works in parallel with the other driver assistancefunctions.

Table 1 shows typical high dynamic range (HDR) exposure and gainsettings for the Aptina MT9M024 1.3 Megapixel camera 12, according to afeature of the present invention. Sensor 6 is set to produce images at45 frames per second (FPS) or 22.2 ms per frame. Image frames 204/206may be partitioned so that the exposures may alternate between a longerexposure setting used primarily for detecting road markings andpedestrian targets, and a shorter exposure which may give better colorinformation and less motion blur to support AHC 24 and TSR 20 driverassistance systems.

TABLE 1 Aptina Grabbing Scheme Sensor running on 45 FPS, 12 bitcompacted image. Algorithm running on two high dynamic range (HDR)images up to 22.5 FPS. Texture Exposure—gain up to 2.75 × 8, integrationperiods up to {14, 0.875, 0.054} ms, knee factors 16, 16. ColorExposure—gain 1, integration periods {11, 0.68, 0.042} ms, knee factors16, 16. Frame capture at constant rate of 45 images per second.Processing Produce 8 bit Tone Mapped image from each of Texture andColor exposures for texture analysis. Work directly on 12 bit images forcolor processing mainly on Color exposure. Night Vision Display the tonemapped image of Texture exposure should be used pedestrian enhancementto be performed on tone mapped image. running as separate processparallel and asynchronous to the main processing.

Reference is now made to FIG. 7 which shows a method 701 according to anaspect of the present invention. Rolling shutter 6 scans line by lineover an image frames 204/206 to read the image data alternately line byline into image frames 204 and 206. In step 703, the image data ofmultiple image frames 204/206 from camera 12 may be received byprocessor 14. In step 705, near infra red (NIR) illumination may beprovided from illuminator 4 to give NIR illumination 4 a forward ofvehicle 18. Illumination 4 a oscillates with an oscillation frequencyand an illumination duty cycle synchronized line by line with theexposure of frames 204/206 of rolling shutter 6.

The term “oscillation” as used herein in the context of near infraredillumination refers to a repetitive variation of illumination intensity.The term “illumination frequency” as used herein is the frequency of theillumination intensity oscillation and the term “illumination period” isthe inverse of the illumination frequency of the oscillation. The term“duty cycle” as used herein duty cycle refers to a substantially squarewave infra-red oscillation between a state of higher illumination and astate of lower illumination. The term “duty cycle” is substantiallyequal to the time spent in the higher state divided by total time.

The term “exposure” as used herein refers to a time period during whicha line of a frame of the rolling shutter is sensitive to light.

Synchronization mechanism 2 and synchronization step 707 may beperformed if camera 12 has a general purpose input output (GPIO) pinwhich can be switched between 1 and 0 with a certain delay after thevertical synch of sensor 6. The signal from the general purpose inputoutput (GPIO) pin may be then sent to the controller of NIR illuminator4 to switch the near infra red illumination on and off synchronouslywith rolling shutter 6. In some cases having the general purpose inputoutput (GPIO) pin may not be available for synchronization 707. In suchcases, NIR illumination controller may be set to a nominal frequency andduty cycle and rolling shutter 6 detects and locks into the exactfrequency and phase. For example if the frequency is 22.5 Hz and theduty cycle is 50%. rolling shutter 6 frequency may be controlled veryprecisely (to an accuracy of 12 ns) by adjusting the horizontal andvertical blanking time of sensor 6.

Alternatively, synchronization (step 707) may be achieved for step 707by maximizing the brightness of a central horizontal strip of an imageframe 204/206.

Reference is now made to FIG. 2 which shows a schematic description ofthe exposure settings with rolling shutter 6, according to a feature ofthe present invention. Two types of image frames 206 and 204 may beshown. In the example, each image row starts and ends exposureapproximately 22 μs after the previous row shown as interval 202. In theexample, the long exposure in frame 204 is 15 milliseconds and theshorter exposure in frame 206 is 11 milliseconds. One important pointmay be that the exposure of the last rows in one image frame, e.g. 206partially overlaps with the first rows of the next image frame, e.g.204.

In both image frames 204 and 206 the frame interval 202 is 22.2 ms.Therefore the first row of the long exposure frame 204 starts 7.2 msafter the start of last row of the short exposure frame 206. The shortexposure frame is 11 ms and the first row starts 11.2 ms after start ofthe last row of the long exposure frame.

Reference is now also made to FIG. 3 which shows schematically graphs ofNIR intensity in percent for NIR illuminator 4 that is on for 15 ms andoff for 30 ms. The on time aligns with the middle row (shown as ahorizontal arrow in FIG. 2) of the long exposure frame 204. This timingwould give very little NIR illumination during the short frame 206 (onlyat the tail end) which may be desirable. However the NIR illuminationduring long frame 204 will also be reduced. There will only be a numberof rows or perhaps even just one row (shown as a horizontal arrow inFIG. 2) which has NIR illumination 4 a for its full length. The issuebecomes more complex when high dynamic range (HDR) sensor may be used.

FIG. 3 also shows pedestrians at 15 m. The pedestrians may be located inthe central horizontal third of the image.

Reference is now made to FIG. 4 a which shows a NIR illuminator 4 thatis on for 22 ms and off for 30 ms, where the on time (indicated byhorizontal arrow) aligns with the middle row of the long exposure frame.The center rows in the image have NIR illumination for their full lengthwith the period of illumination dropping at the top and bottom. Howeverthere may be significant NIR illumination during the exposure time ofthe top and bottom row of short exposure frame 206.

Below is Table 2 which shows typical different DAS applications and theillumination requirements:

TABLE 2 Vehicle Detection: Vehicle detection at night at long distancesby detecting headlights and taillights NIR illumination may bedetrimental to assessing taillight color. Close vehicle detection maybenefit from NIR illumination but NIR illumination may be not required.Pedestrian (and animals) Detection 23: up to 20 m pedestrian cantypically be detected well with the host vehicles low beams. Beyond thatrange NIR illumination may be recommended. Traffic sign recognition 20:Regular reflective traffic signs may be typically located at the side ofthe road and infrequently overhead. Electronic signs may be most oftenlocated overhead. For correct color recognition and to avoidover-saturation it may be important that the reflective signs may be notilluminated by the NIR 4 during the recognition stage. Thus it may beimportant the central strip of the image not be illuminated by NIRduring the short exposure frame 206 and ideally neither the top. Dark,distant signs may benefit from NIR illumination during the long exposureframe. Automatic headlight control 24: Oncoming headlights and precedingtaillights may be detected without any host vehicle illumination. NIRillumination may be detrimental to distinguishing, using color, betweenreflectors at the side of the road and distant taillights. It may beimportant the central strip of the image not be illuminated by NIRduring at least one frame. Lane departure warning 21: LDW can benefitfrom NIR illumination for distance road markings. In order to determinelane mark color it would be best to not have NIR illumination in thelower region of the image which may be well illuminated by the lowbeams.

The above requirements in Table 2, lead to an illuminator 4 design whichspreads the energy vertically over +/−5 deg and horizontally +/−20degrees.

FIG. 4 b shows an NIR spatial profile 60 overlaid on an image withrespective percentage (%) NIR intensities for frames 204/206.

Reference is now made to FIG. 5 which shows schematically a typicalspatial distribution 60 in image space coordinates of NIR illuminationcone, according to a feature of the present invention. The profile maybe not optimal as its horizontal spread should be greater. It may bemost interesting to look at how such an illumination profile matches thesensor timing.

Reference is now made to FIG. 6 which shows the NIR spatial profile 60overlaid on an image, according to an feature of the present invention.By carefully aligning the row timing and the NIR profile 60 maximumillumination on the important regions of the image frames during thelong exposure frame 204 and virtually no illumination during the top andbottom rows of the short exposure image frame 206. Thus the shortexposure image frame 206 may be almost clean of NIR artifacts. As anadded bonus, the flickering of a reflector between short and longexposures 206 and 204 respectively, can further aid in distinguishingbetween reflectors at the side of the road and distant taillights.

Effects of High Dynamic Range (HDR) Imagers

Consider the high dynamic range (HDR) integration settings for the longexposure frame shown in Table 1. The first integration period of 14 msis followed by integration period of 0.845 ms and then 0.054 ms.Typically a very bright patch in the image will saturate the long andmedium integration periods and the pixel value will be determined by theshort integration period. Less bright patches will perhaps saturateduring the long integration period and the pixel value will bedetermined by the medium integration period. This high dynamic range(HDR) method is not designed for flickering light sources and has thefollowing behavior: for patches that are saturated at the longintegration period but then have no response during the medium and shortintegration periods, the pixel has a maximum value of about 256 (in 12bit image with a maximum of 4095). For patches that are saturated at themedium integration period but then have no response during the shortintegration period, the pixel has a maximum value of about 1024.Referring again to FIGS. 4 a and 4 b, the bottom rows in the image theNIR illumination will not continue till the end of the frame. This meansthat for a few rows there will be only partial illumination during theshort integration. Then for about 37 rows there will be no illuminationduring the short integration period and for only part of the mediumintegration period. Then for about 292 rows there will not beillumination during the short or medium integration periods. For theselines the maximum pixel value on reflectors will be limited to a maximumof 256.

Pedestrian Detection

Consider now a pedestrian illuminated by the NIR beams. Clothing, evenblack clothing, may be often very reflective to NIR and can appearsaturated in the image. However due to the effect described above, thebottom part of the pedestrian might not reach above a value of 256. Thenthere will be a band with a value of 1024 and only the top part willreach the true brightness levels above 1024. This makes pedestriandetection difficult as the pedestrian will look different depending onthe exact vertical location in the image. It will also look bad in thedisplay. These two issues may be addressed separately.

Pedestrian detection can be divided into four steps: (1) quickly findinginitial candidates (2) more expensive pattern recognition often using anexample based classifier (3) tracking over time and accumulatingclassification score (4) final approval or rejection. Each may beaffected differently by the NIR and high dynamic range (HDR) timing:

-   1. Finding initial candidates: This first step often uses vertical    and horizontal edges. The vertical edges, based on horizontal    derivatives, are significantly not affected be the timing issue. The    horizontal derivatives of the two lines, above and below the    transition should be computed without vertical smoothing. The    horizontal edges can be computed as usual but suppressing fake edges    at the transition rows if the lower row value may be at the relevant    saturation level (256 or 1024). The high dynamic range (HDR) effect    can be used as a rejection cue since any pixel below the 1024 or 256    cutoff lines and with values above 1024 or 256 respectively must be    a light source and not a reaction.-   2. Classifier: The classifier cannot work directly on the images    where the lower part is clipped at 256 and 1024 since the cutoff    line will hit each pedestrian at a different vertical point on the    body. One may simply clip the whole patch to be classified at 256    but then useful information might be lost. An alternative may be to    leave the top half of the patch unchanged and the lower half (which    includes both the 1024 and the 256 cutoff lines) is then clipped    to 256. This gives uniform images but allows for full detail at    least in the upper part of the body.-   3. Tracking: Tracking may be performed on the patch and image after    is clipped to 256.-   4. Final approval: This stage combines information from previous    stages and does not address the image directly.    Night Vision Display

According to an aspect of the present invention the near infra redillumination may be switched on and off in alternating frames 204/206.Multiple NIR illuminated frames 204 may be provided to processor 14which may be used by pedestrian detection 23 for example. Multiplenon-NIR illuminated frames 206 may be provided to processor 14 which maybe used by AHC 24 and TSR 20 driver assistance systems for example.Partitioning of frames 204/206 into either NIR illuminated 204 or nonNIR illuminated frames 206 may also alternate between a longer or ashorter sensor 6 exposure time. The shorter sensor 6 time may be usefulto give better color information and less motion blur when compared withthe longer exposure time of sensor 6.

A consequence of switching the NIR illumination on and off in alternateframes may be that for a night vision display, only half the frames maybe usable and the effective frame rate may be halved. If 20 Hz is theminimum acceptable frame rate for night vision display then theacceptable frame rate must be 40 Hz or higher. If instead of switchingoff the NIR illuminator 4, the power is reduced to 50% every frame 206,the low reduced NIR power frame 206 can still be used for night visiondisplay. The power values cannot simply be multiplied by a factor of twosince not all the image brightness may be due to the NIR illumination.Some of the image brightness may be from actual light sources and thelower part of image frames 204/206 may also be illuminated by low beamsof host vehicle 18.

The first step may be to calibrate the low beams versus the NIRilluminator 4 to determine a smooth image function that describes bywhat value may be needed to multiply that particular pixel when the NIRillumination may be halved. This calibration may be done automaticallywhen vehicle 18 may be stationary. During driving the 50% frame 206 maybe multiplied by a factor α. Pixels which are bright and suspected ofbeing a light source may be tracked between consecutive frames 204/206by first assuming a scaling of α and secondly using a scaling of 1 for alight source. The match that gives the best tracking score.

The above discussion assumed that the exposures are linear and remainconstant between consecutive frames. If a high dynamic range (HDR)sensor may be used then the image values can be mapped to a linear imagebefore scaling. If alternating exposure values are used then scalingvalues may be adapted accordingly.

The near infra red (NIR) and high dynamic range (HDR) timing produce twohorizontal cutoff lines in the image. In order to produce a smoothtransition, pedestrians and other regions may be treated separately. Forpedestrians, if the upper part of the pedestrian, just above the cutoffline is above the 1024 cutoff value, first compute an average value forall pixels of 1024 and up. Then all pixels are colored of the detectedpedestrian that are below the cutoff line, that are of the cutoff valuesof 256 or 1024 according to this average pixel value. A similartechnique may be used to blend the pixels below the lower cutoff line ifneed be.

For non pedestrian regions a similar idea may be used, however:

-   (1) We only extend it to 100 rows below the upper cutoff line with a    linear transition between the average value and the cutoff value of    256.-   (2) We only affect pixels that are connected to pixels above the    cutoff line which are above the cutoff value. This may be done using    a recursive pixel fill (or paint) operation.    Switching the Near Infra Red (NIR) Source on Each Odd Row

If LED illuminators 4 are used then can be switched on and off at veryhigh frequencies. Consider what would happen if during the 22 ms whenthe near infra red (NIR) is typically on, the near infra red (NIR)source was switched off during the short integration period of the oddrows. If the short integration period matched the shift between starttimes of consecutive rows then reflectors would not reach values above1024 on odd rows. Thus it would be simple to tell if a bright patch wasa light source or a reflector: it would have even rows reaching levelsabove 1024 and odd rows clipped at 1024.

Close and mid-range pedestrians will often reach values above 1024. Fornear infra red (NIR) display, if their is a pixel on an odd row with avalue of 1024 and higher values above and below, the 1024 may bereplaced with the average value. It may be possible do the same forpedestrian detection: the edge detection for initial candidates willstill work well and so will a classifier if the classifier is trained toexamples that were processed in a similar manner.

The indefinite articles “a”, “an” is used herein, such as “anilluminator”, “a driver assist system” have the meaning of “one or more”that is “one or more illuminators” or “one or more driver assistsystems”.

Examples of various features/aspects/components/operations have beenprovided to facilitate understanding of the disclosed features of thepresent invention. In addition, various preferences have been discussedto facilitate understanding of the disclosed features of the presentinvention. It is to be understood that all examples and preferencesdisclosed herein are intended to be non-limiting.

Although selected features of the present invention have been shown anddescribed individually, it is to be understood that at least aspects ofthe described features may be combined. Also although selected featuresof the present invention have been shown and described, it is to beunderstood the present invention is not limited to the describedfeatures. Instead, it is to be appreciated that changes may be made tothese features without departing from the principles of the invention,the scope of which is defined by the claims and the equivalents thereof.

We claim:
 1. A system mountable in a motor vehicle, the systemcomprising: a camera; a processor configured to receive image data fromthe camera; wherein the camera includes a rolling shutter configured tocapture said image data by scanning during a frame period and to readthe image data into a plurality of image frames, wherein the processorincludes a plurality of driver assistance modules configured to performrespectively a plurality of driver assistance functions; wherein saidcamera is configured to provide to the processor said image frames,wherein the image frames are partitioned into first image frames for useby a first driver assistance module and into second image frames for useby a second driver assistance module; a near infra-red illuminatorconfigured to provide a near infra-red illumination cone in the field ofview of the camera, wherein the near infrared illumination oscillateswith an illumination period; wherein the first driver assistance moduleis configured to use the near infra-red illumination in a part of animage in at least one of the first image frames, wherein the seconddriver assistance module is configured to use a part of an image in atleast one of the second image frames that does not have said nearinfrared illumination; a synchronization mechanism configured tosynchronize said illumination period to said frame period of saidrolling shutter to time said near infra-red illumination to be in saidpart of the image of said at least one of the first image frames and toavoid said near infra-red illumination in said part of the image of saidat least one of the second image frames.
 2. The system according toclaim 1, wherein the image frames include a long exposure frame and ashort exposure frame, wherein said synchronization mechanism isconfigured to turn on the near infra-red illumination during exposure ofat least one of the middle rows of said long exposure frame.
 3. Thesystem according to claim 1, wherein said frame period is selected sothat said synchronization mechanism provides a spatial profile of saidnear infra-red illumination cone substantially aligned vertically to aspecific region of the image frame.
 4. The system according to claim 1,wherein a parameter of said camera is configurable to alternaterespective values between the first and second image frames.
 5. Thesystem according to claim 4, wherein said parameter is exposure periodof said rolling shutter.
 6. The system according to claim 1, whereinsaid first driver assistance module is configured to perform pedestriandetection responsive to said near infra-red illumination in said firstimage frames.
 7. The system according to claim 1, wherein said seconddriver assistance module uses color information from said second imageframes to detect headlights of an oncoming vehicle in the field of viewof the camera.
 8. The system according to claim 1, wherein said seconddriver assistance module uses color information from said second imageframes to detect taillights of a leading vehicle in the field of view ofthe camera.
 9. The system according to claim 1, wherein said seconddriver assistance module uses color information from said second imageframes to detect a traffic sign in the field of view of the camera. 10.A method operable in a system mountable in a motor vehicle, the systemincluding: a processor, a camera having a rolling shutter configured tocapture image data by scanning during a frame period and to read theimage data into a plurality of image frames, wherein the processorincludes a plurality of driver assistance modules configured to performrespectively a plurality of driver assistance functions, the methodcomprising: receiving a plurality of image frames from the camera;partitioning the images frames into first image frames for use by saidfirst driver assistance module and into second image frames for use by asecond driver assistance module; providing near infra-red illuminationin the field of view of the camera, wherein the near infraredillumination oscillates at an illumination frequency with anillumination period; configuring a first driver assistance module to usethe near infra-red illumination in a part of an image in at least one ofthe first image frames; configuring a second driver assistance module touse a part of an image in at least one of the second image frames thatdoes not have said near infrared illumination; and synchronizing saidillumination period to said frame period of said rolling shutter to timesaid near infra-red illumination to be in said part of the image of saidat least one of the first image frames and to avoid said near infra-redillumination in said part of the image of said at least one of thesecond image frames.
 11. The method of claim 10, wherein the imageframes include a long exposure frame and a short exposure frame, whereinsaid synchronizing turns on the near infra-red illumination duringexposure of at least one of the middle rows of said long exposure frame.12. The method of claim 10, further comprising: alternating values of acamera parameter respectively in the alternating first and second imageframes.
 13. The method of claim 12, wherein said camera parameter isexposure period of said rolling shutter.